1. Field of the Invention
The present invention generally relates to evaluation of scoring models and, more particularly, to the use of ranking-based measures to evaluate the performance of regression models. The invention has particular application to evaluation of prediction models with regard to the ranking of customers and/or potential customers according to their potential to spend for goods and services.
2. Background Description
Evaluating prediction models of customers according to their potential to spend has been done through residual-based measures; i.e., the difference between the predicted and actual spending by some known customers. This approach suffers from two main drawbacks: (1) it is non-robust to outliers (for example, gross errors in the data used for evaluation), and (2) it is not the appropriate measure if the goal is just to identify the best prospective customers.
The standard approach to evaluating regression models on holdout data is through additive, residual-based loss functions, such as squared error loss or absolute loss. These measures are attractive from a statistical perspective as they have likelihood interpretations and because, from an engineering or scientific perspective, they often represent the “true” cost of the prediction errors.
Other approaches to regression model evaluation include Regression Error Curves, where the model is evaluated according to its error rate at different levels of “error tolerance”; and using medians of the absolute deviations (MAD), rather than their mean, as the error measure:MAD=Median(|r1|, . . . , |rn|)  (1)
There are many companies with relatively small wallet size and a few companies with very large wallet size. Therefore, evaluation measures such as mean squared error and mean absolute error can be greatly influenced by a small subset of companies that have very large wallets and for which the models are more likely to make larger absolute errors. On the other hand, measures such as median squared error can completely ignore the performance of the model on the companies with large IT wallet size, which are usually the most important customers. An approach that is often used to mitigate the effects of a skewed distribution (especially in modeling) is to transform the numbers to a logarithmic scale. This approach, however, is not adequate for the evaluation of prediction models, since log-dollars is a unit that does not have a clear financial meaning and, therefore, cannot be used in conjunction with other financial variables such as budget and costs.